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DETAILED ACTION 

In response to Applicant's amendnnents/rennarks filed 02/23/2009 and telephone 
interviewed on 05/29/2009, the examiner's amendment was authorized by attorney of 
record, Michael J. Mallie, Attorney for Applicants. 

> Claims 1, 5, 8, 10-13, 17, 20, 22-25, 29, 32, 34-36, 46-47 and 49 are currently 
amended [see Examiner's Amendments cited below for details]. 

> Claims 44-45 are currently canceled. 

> Claims 2-3, 6-7, 14-15, 18-19, 26-27, 30-31 and 37-43 were previously canceled. 

> Claims 4, 9, 16,21, 28, 33, 48, and 50-52 were previously presented. 

Should the changes and/or additions be unacceptable to applicant, an 
amendment may be filed as provided by 37 CFR 1.312. To ensure consideration of 
such an amendment, it MUST be submitted no later than the payment of the issue fee. 

EXAMINER'S AMENDMENT 

The application has been amended as follows: 
In the Claims: 

> replace the claims with the following claim set: 
1. (Currently Amended) A method comprising: 

reproducing a paper document via a document reproduction system: 
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creating a nnultinnedia annotation for [[a]]the paper document during tlie 
reproduction , wlierein tlie multimedia annotation r e pr e s e nt i ng at le ast on e of is_an audio 
sound and a v i d e o c li p that is input by a user via a microphone of the document 
reproduction svstem :[[and]] 

creating a first multimedia document by combining the paper document and the 
multimedia annotation , wherein the first multimedia document includes r e pr e s e nt e d by a 
first bar code e ncod i ng that encodes the at le ast on e of th e audio sound therein: af(4 
v i deo c li p, 

wh e r ei n th e f i rst mu l t i m e d i a docum e nt i s g e n e rat e d i n r e spons e to a us e r 
i n i t i ated copy request to reproduce the paper document v i a a document reproduct i on 
syst e m, wh e r ei n th e mu l t i m e d i a annotat i on i s captur e d v i a a m i crophon e of an i nput 
d e v i c e of th e docum e nt r e product i on syst e m wh ile th e pap e r docum e nt i s b ei ng 
r e produc e d v i a th e docum e nt r e product i on syst e m, wh e r ei n th e captur e d mu l t i m e d i a 
annotat i on i s d i g i t i z e d by th e docum e nt r e product i on syst e m and e ncod e d w i th i n th e 
f i rst bar cod e , and 

wh e r ei n processing the first multimedia document ;, wh i ch wh e n scann e d by a 
proc e ss, th e proc e ss caus e s th e pr i nt e d mu l t i m e d i a annotat i on to b e d e cod e d, th e at 
le ast on e of th e aud i o sound and v i d e o c li p to b e e xtract e d from th e mu l t i m e d i a 
annotat i on, and th e le ast on e e xtract e d aud i o sound and v i d e o c li p can b e p l ay e d v i a a 
mu l t i m e d i a p l ay e r- 
decoding the multimedia annotation from the first bar code: 
extracting the audio sound: and 
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playing the audio sound via a nnultinnedia player. 
2-3. (Cancelled) 

4. (Previously Presented) The method of claim 1, wherein a location indicator 
associated with the multimedia annotation is placed on the first multimedia document, 
wherein the location indicator indicates where the multimedia annotation can be 
retrieved and played. 

5. (Currently Amended) The method of claim 4, wherein the location indicator 
comprises a first Uniform Resource Locator (URL) indicated in plain text .-af(4 wherein 
the first multimedia document includes a second bar code, wh e r ei n th e f i rst URL i s 
i nd i cat e d i n p l a i n t e xt, and wherein the second bar code represents the first URL in an 
encrypted form. 

6-7. (Canceled) 

8. (Currently Amended) The method of claim 1 , further comprising: 

generating an image of the paper document, wherein the image of the paper 
document feetfi§-is_unconsciously captured via the document reproduction system 
during the reproduction of the paper document without user intervention; 
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creating a second nnultinnedia document by combining tlie image of tlie paper 
document and tlie multimedia annotation; and 

storing tlie i mag e of tli e pap e r docum e nt and tli e mu l t i m e d i a annotat i on second 
multimedia document in a storage that stores a plurality of multimedia documents , 
wherein the second multimedia document is an electronic document associated with the 
first multimedia document which is a physical document. 

9. (Previously Presented) The method of claim 8, wherein the second multimedia 
document is represented as a second Uniform Resource Locator (URL) printed on the 
first multimedia document, and wherein the image of the paper document and the 
multimedia annotation is accessed with the second URL. 

1 0. (Currently Amended) The method of claim 9, wherein the first multimedia 
document includes a third bar code that i s us e d to represents the second URL. 

1 1 . (Currently Amended) The method of claim 8, further comprising: 

automatically s e nd i ng emailing the second multimedia document to a recipient fey 
ele ctron i c ma il as a part of during the r e produc i ng reproduction of the paper document 
v i a th e docum e nt r e product i on syst e m , wherein the recipient is specified by a user via 
an interface of the document reproduction system when the user reproduces the paper 
document using the document reproduction system. 
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12. (Currently Amended) The method of claim 1 1 , wherein the recipient receives the 
i mag e of th e pap e r docum e nt and th e mu l t i m e d i a annotat i on second multimedia 
document in the form of Multi-purpose Internet Mail Extension (MIME). 

13. (Currently Amended) A machine-readable medium providing instructions[[,]] which^ 
when executed by a set of one or more processors, cause said set of one or more 
processors to perform the following: 

reproducing a paper document via a document reproduction system; 

creating a multimedia annotation for [[a]]the paper document during the 
reproduction , wherein the multimedia annotation r e pr e s e nt i ng at le ast on e of is_an audio 
sound and a v i d e o c li p that is input by a user via a microphone of the document 
reproduction system :[[ and]] 

creating a first multimedia document by combining the paper document and the 
multimedia annotation , wherein the first multimedia document includes r e pr e s e nt e d by a 
first bar code e ncod i ng that encodes the at le ast on e of th e audio sound therein: aft4 
v i deo c li p, 

wh e r ei n th e f i rst mu l t i m e d i a docum e nt i s g e n e rat e d i n r e spons e to a us e r 
i n i t i ated copy roqucGt to reproduce the paper document v i a a document reproduct i on 
syst e m, wh e r ei n th e mu l t i m e d i a annotat i on i s captur e d v i a a m i crophon e of an i nput 
d e v i c e of th e docum e nt r e product i on syst e m wh ile th e pap e r docum e nt i s b ei ng 
r e produc e d v i a th e docum e nt r e product i on syst e m, wh e r ei n th e captur e d mu l t i m e d i a 
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annotat i on i s d i g i t i z e d by th e docum e nt r e product i on syst e m and e ncod e d w i th i n th e 
f i rst bar cod e , and 

wh e r ei n processing the first multimedia document ;, wh i ch wh e n scann e d by a 
proc e ss, th e proc e ss caus e s th e pr i nt e d mu l t i m e d i a annotat i on to b e d e cod e d, th e at 
le ast on e of th e aud i o sound and v i d e o c li p to b e e xtract e d from th e mu l t i m e d i a 
annotat i on, and th e le ast on e e xtract e d aud i o sound and v i d e o c li p can b e p l ay e d v i a a 
mu l t i m e d i a p l ay e r- 
decoding the multimedia annotation from the first bar code: 
extracting the audio sound: and 
playing the audio sound via a multimedia player. 

14-15. (Cancelled) 

16. (Previously Presented) The machine-readable medium of claim 13, wherein a 
location indicator associated with the multimedia annotation is placed on the first 
multimedia document, wherein the location indicator indicates where the multimedia 
annotation can be retrieved and played. 

17. (Currently Amended) The machine-readable medium of claim 16, wherein the 
location indicator comprises a first Uniform Resource Locator (URL) indicated in plain 
text.-aft4 wherein the first multimedia document includes a second bar code, wh e r ei n 
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t7-and wherein tlie second bar code represents tlie 



first URL in an encrypted form. 
18-19. (Cancelled) 

20. (Currently Amended) The machine-readable medium of claim 13, further 
comprising: 

generating an image of the paper document, wherein the image of the paper 
document feetf»§-is_unconsciously captured via the document reproduction system 
during the reproduction of the paper document without user intervention; 

creating a second multimedia document by combining the image of the paper 
document and the multimedia annotation; and 

storing the i mag e of th e pap e r docum e nt and th e mu l t i m e d i a annotat i on second 
multimedia document in a storage that stores a plurality of multimedia documents , 
wherein the second multimedia document is an electronic document associated with the 
first multimedia document which is a physical document. 

21 . (Previously Presented) The machine-readable medium of claim 20, wherein the 
second multimedia document is represented as a second Uniform Resource Locator 
(URL) printed on the first multimedia document, and wherein the image of the paper 
document and the multimedia annotation is accessed with the second URL. 
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22. (Currently Amended) The machine-readable medium of claim 21 , wherein the first 
multimedia document includes a third bar code that i s us e d to represents the second 
URL. 

23. (Currently Amended) The machine-readable medium of claim 20, further 
comprising: 

automatically s e nd i ng emailing the second multimedia document to a recipient fey 
ele ctron i c ma il as a part of during the r e produc i ng reproduction of the paper document 
v i a th e docum e nt r e product i on syst e m , wherein the recipient is specified by a user via 
an interface of the document reproduction system when the user reproduces the paper 
document using the document reproduction system. 

24. (Currently Amended) The machine-readable medium of claim 23, wherein the 
recipient receives the i mag e of th e pap e r docum e nt and th e mu l t i m e d i a annotat i on 
second multimedia document in the form of Multipurpose Internet Mail Extension 
(MIME). 

25. (Currently Amended) A computer system, comprising: 

a bus; 

a data storage device coupled to the bus; and 
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a processor coupled to the data storage device, tlie processor operable to 
receive instructions which, when executed by the processor, cause the processor to 
perform a method comprising: 

reproducing a paper document via a document reproduction system; 

creating a multimedia annotation for [[a]]the paper document during the 
reproduction , wherein the multimedia annotation r e pr e s e nt i ng at le ast on e of is 
an audio sound and a v i d e o c li p that is input by a user via a microphone of the 
document reproduction system :[[ and]] 

creating a first multimedia document by combining the paper document 
and the multimedia annotation , wherein the first multimedia document includes 
r e pr e s e nt e d by a first bar code e ncod i ng that encodes the at le ast on e of th e 
audio sound therein: and v i d e o c li p. 

wh e r ei n th e f i rst mu l t i m e d i a docum e nt i s g e n e rat e d i n r e spons e to a us e r 
i n i t i ated copy request to reproduce the paper document v i a a document 
r e product i on syst e m, wh e r ei n th e mu l t i m e d i a annotat i on i s captur e d v i a a 
m i crophon e of an i nput d e v i c e of th e docum e nt r e product i on syst e m wh ile th e 
pap e r docum e nt i s b ei ng r e produc e d v i a th e docum e nt r e product i on syst e m, 
wh e r ei n th e captur e d mu l t i m e d i a annotat i on i s d i g i t i z e d by th e docum e nt 
r e product i on syst e m and e ncod e d w i th i n th e f i rst bar cod e , and 

wh e r ei n processing the first multimedia document ;, wh i ch wh e n scann e d 
by a proc e ss, th e proc e ss caus e s th e pr i nt e d mu l t i m e d i a annotat i on to b e 
d e cod e d, th e at le ast on e of th e aud i o sound and v i d e o c li p to b e e xtract e d from 
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th e nnu l t i nn e d i a annotat i on, and th e le ast on e e xtract e d aud i o sound and v i d e o 

c li p can bo p l ayed v i a a mu l t i med i a p l ayer- 
decoding the multimedia annotation from the first bar code: 
extracting the audio sound: and 
playing the audio sound via a multimedia player. 

26-27. (Cancelled) 

28. (Previously Presented) The computer system of claim 25, wherein a location 
indicator associated with the multimedia annotation is placed on the first multimedia 
document, wherein the location indicator indicates where the multimedia annotation can 
be retrieved and played. 

29. (Currently Amended) The computer system of claim 28, wherein the location 
indicator comprises a first Uniform Resource Locator (URL) indicated in plain text .-aft4 
wherein the first multimedia document includes a second bar code, wh e r ei n th e f i rst 
URL i s i nd i cat e d i n p l a i n t e xt, and wherein the second bar code represents the first URL 
in an encrypted form. 



30-31. (Canceled) 
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32. (Currently Amended) The computer system of claim 25. furth e r compr i s i ng wherein 
the method performed by the processor further comprises : 

generating an image of the paper document, wherein the image of the paper 
document feetf»§-is_unconsciously captured via the document reproduction system 
during the reproduction of the paper document without user intervention; 

creating a second multimedia document by combining the image of the paper 
document and the multimedia annotation; and 

storing the i mag e of th e pap e r docum e nt and th e mu l t i m e d i a annotat i on second 
multimedia document in a storage that stores a plurality of multimedia documents , 
wherein the second multimedia document is an electronic document associated with the 
first multimedia document which is a physical document. 

33. (Previously Presented) The computer system of claim 32, wherein the second 
multimedia document is represented as a second Uniform Resource Locator (URL) 
printed on the first multimedia document, and wherein the image of the paper document 
and the multimedia annotation is accessed with the second URL. 

34. (Currently Amended) The computer system of claim 33. wherein the first 
multimedia document includes a third bar code that i s us e d to represents the second 
URL. 
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35. (Currently Amended) The computer system of claim 32. furth e r compr i s i ng wherein 
the method performed by the processor further comprises: 

automatically s e nd i ng emailing the second multimedia document to a recipient fey 
ele ctron i c ma il as a part of during the r e produc i ng reproduction of the paper document 
v i a th e docum e nt r e product i on syst e m , wherein the recipient is specified by a user via 
an interface of the document reproduction system when the user reproduces the paper 
document using the document reproduction system. 

36. (Currently Amended) The computer system of claim 35, wherein the recipient 
receives the i mag e of th e pap e r docum e nt and th e mu l t i m e d i a annotat i on second 
multimedia document in the form of Multi-purpose Internet Mail Extension (MIME). 

37-45. (Cancelled) 

46. (Currently Amended) The method of claim [[45]]1, wherein the microphone is 
automatically activated when the user selects a reproduction function of the document 
reproduction system to reproduce the paper document. 

47. (Currently Amended) The method of claim 8, further comprising: 

capturing a video clip of the multimedia annotation from [[a li the user us i ng via a 
video camera of the i nput d e v i c e document reproduction system when the user 
reproduces the paper document using the document reproduction system. 
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48. (Previously Presented) Tlie nnetliod of claim 47, wherein the video camera is 
automatically activated when the user selects a reproduction function of the document 
reproduction system to reproduce the paper document. 

49. (Currently Amended) The method of claim 8, further comprising[[,]]: 

in response to a request to retrieve a multimedia document from among the 
plurality of multimedia documents stored in the storage, performing a content-based 
search on multimedia annotations of the plurality of multimedia documents for the 
requested multimedia document within the storage based on content of the multimedia 
annotation associated with the requested multimedia document. 

50. (Previously Presented) The method of claim 49, wherein the content-based search 
is performed via an optical character recognition (OCR) process on the multimedia 
annotation of the multimedia documents being searched. 

51 . (Previously Presented) The method of claim 49, wherein the content-based search 
is performed via a speech recognition mechanism on an audio sound of the multimedia 
annotation of the multimedia documents being searched. 
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52. (Previously Presented) Tlie nnetliod of claim 49, wherein the content-based search 
is performed on a video clip of the multimedia annotation based on an image of the user 
using face recognition techniques. 



Allowable Subject Matter 

The prior art made of record: 

> Morris etal., US005420974A filed 10/15/1992 

> Hodaetal., US 004831610 filed 03/03/1987 

> Schenaetal., US006448979B1 filed 01/25/1999 

> Hallidayetal., US005880740A filed 06/12/1996 



❖ Claim(s) 1, 4-5, 8-13, 16, 20-25, 28-29, 32-36 and 46-52, are allowed: 
The following is a statement of reasons for the indication of allowable 
subject matter: 

Interpreting the claims in light of the specification. Examiner finds the claimed 
invention is patentably distinct from the prior art of record, which set forth in the 
followings: 

Under the broadest reasonable interpretation of the claimed limitation which is 
consistence with the Applicant's Specification. The prior art cited above fails to teach all 
of the Applicant's claimed limitation. In particularly, the claimed invention 
advantageously provides a finer level of detail that enables a multimedia annotation 
being annotated during the reproducing of a paper document; wherein the multimedia is 
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an audio sound tliat is input by a user via a microplione of tlie document reproduction 
system; and creating a first multimedia document by combining tlie paper document 
and tlie multimedia annotation, wherein the first multimedia document includes a first 
bar code that encodes the audio sound therein; processing the first multimedia 
document, this allows the decoding the multimedia annotation from the first bar code; 
extracting the audio sound; and playing the audio sound via a multimedia player [see 
the currently amended claim(s) 1,13 and 25 which cited above and the Applicant's 
remarks filed 02/23/2009 at page(s) 16-17 the last paragraph and supported by the 
current specification at page 8 line 25 through page 9 line 2.] 

The Examiner asserts that the claims overcome the prior art of record as 
describes above when the limitations are read in combination with the respective 
claimed limitations in their entirety. 

Any comments considered necessary by applicant must be submitted no later 
than the payment of the issue fee and, to avoid processing delays, should preferably 
accompany the issue fee. Such submissions should be clearly labeled "Comments on 
Statement of Reasons for Allowance." 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Quoc A. Iran whose telephone number is 571-272- 
8664. The examiner can normally be reached on Mon through Fri 8AM - 5PM. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Doug Hutton can be reached on (571)272-4137. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 

Patent Application Information Retrieval (PAIR) system. Status information for 

published applications may be obtained from either Private PAIR or Public PAIR. 

Status information for unpublished applications is available through Private PAIR only. 

For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 

you have questions on access to the Private PAIR system, contact the Electronic 

Business Center (EBC) at 866-217-9197 (toll-free). 

/Quoc A. Iran/ 
Examiner, Art Unit 2176 



/DOUG HUTTON/ 

Supervisory Patent Examiner, Art Unit 2176 



